Information-processing apparatus, information-processing methods, recording mediums, and programs

ABSTRACT

An information-processing apparatus including a reproducer for synchronously reproducing content data common to another information-processing apparatus, a user information input for inputting first user condition information including a voice and image of a user, a user information receiver for receiving second user condition information including a voice and image of an other user from the other information-processing apparatus, a synthesizer for synthesizing a voice and image of the content data synchronously reproduced by the reproducer with the voice and image of the second user condition information received by the user-information receiver; an information analysis unit for analyzing at least either the first user condition information input or the second user condition information; and a parameter-setting unit for setting a control parameter used for controlling a process on the basis of an analysis result produced by the information analysis unit.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.11/179,560, filed Jul. 13, 2005, and is based upon and claims thebenefit of priority from the prior Japanese Patent Application No. JP2004-218532 filed on Jul. 24, 2004, the entire contents of each of whichare incorporated herein by reference.

BACKGROUND OF THE INVENTION

The present invention relates to information-processing apparatus,information-processing methods, recording mediums, and programs. Moreparticularly, the present invention relates to information-processingapparatus, information-processing methods, programs, and recordingmediums, which are connected to each other by a network, used forsynthesizing a content common to the apparatus with voices and images ofusers operating the apparatus and used for reproducing a analysis resultsynchronously.

The apparatus in related art used in interactions with people atlocations remotely separated from each other include the telephone, theso-called TV telephone, and a video conference system. There is also amethod whereby personal computers or the like are connected to theInternet and used for chats based on texts and video chats based onimages and voices. Such interactions are referred to hereafter as remotecommunications.

In addition, there has also been proposed a system wherein persons eachcarrying out remote communications with each other share a virtual spaceand the same contents through the Internet by using personal computersor the like connected to the Internet. For more information on such asystem, refer to documents such as Japanese Patent Laid-open No.2003-271530.

SUMMARY OF THE INVENTION

In the method in related art allowing users at locations remotelyseparated from each other to share the same content, however, the userscommunicate with each other by transmission of mainly informationwritten in a language. Thus, the method in related art has a problem ofdifficulties to express the mind and situation of a user to another userin comparison with the face-to-face communication in which the user isactually facing the communication partner.

In addition, the method in related art whereby the user can view animage of the communication partner and listen to a voice of the partneralong with the same content shared with the partner has a problem ofdifficulties to operate the apparatus in order to optimally synthesizethe image and voice of the partner with the image and sound of thecontent by manual operations or the like, which are carried out by theuser, due to complexity of the apparatus.

Addressing the problems described above, inventors of the presentinvention have devised a technique capable of setting a synthesis of aplurality of images and a plurality of sounds with ease in accordancewith the conditions of users in a process carried out by the users toview and listen to the same content at locations remote from each other.

In accordance with an embodiment of the present invention, there isprovided an information-processing apparatus including:

reproduction means for reproducing content data common to theinformation-processing apparatus and the other information-processingapparatus synchronously with the other information-processing apparatus;

user information input means for inputting first user conditioninformation including a voice and image of a user operating theinformation-processing apparatus;

user information receiver means for receiving second user conditioninformation including a voice and image of an other user from the otherinformation-processing apparatus operated by the other user;

synthesis means for synthesizing a voice and image of the content datasynchronously reproduced by the reproduction means with the voice andimage of the second user condition information received by theuser-information receiver means;

information analysis means for analyzing at least either the first usercondition information input by the user-information input means or thesecond user condition information received by the user-informationreceiver means; and

parameter-setting means for setting a control parameter used forcontrolling a process, which is carried out by the synthesis means tosynthesize voices and images, on the basis of an analysis resultproduced by the information analysis means.

In accordance with an embodiment of the present invention, it is alsopossible to provide a configuration in which the first user conditioninformation also includes information on the environment of the userwhereas the second user condition information also includes informationon the environment of the other user.

The information-processing apparatus according to an embodiment of thepresent invention further includes analysis-object determination meansfor generating a determination result as to whether or not the seconduser condition information received by the user-information receivermeans is to be analyzed by the information analysis means and, if thedetermination result produced by the analysis-object determination meansindicates that the second user condition information is not to beanalyzed by the information analysis means, the information analysismeans analyzes only the first user condition information.

The information-processing apparatus according to an embodiment of thepresent invention further includes output means for outputting a voiceand an image, which are obtained as a synthesis result carried out bythe synthesis means, wherein the parameter-setting means sets a controlparameter of the output means on the basis of an analysis result carriedout by the information analysis means.

It is also possible to provide a configuration in which theparameter-setting means sets a control parameter of an electronicapparatus, which is connected to the information-processing apparatus,on the basis of an analysis result carried out by the informationanalysis means.

It is also possible to provide a configuration in which theparameter-setting means sets a control parameter of the otherinformation-processing apparatus on the basis of an analysis resultcarried out by the information analysis means, and

the information-processing apparatus further includes sender means fortransmitting the control parameter set by the parameter-setting means tothe other information-processing apparatus.

The information-processing apparatus according to an embodiment of thepresent invention further includes acceptance determination means forgenerating a determination result as to whether or not a controlparameter transmitted by the other information-processing apparatus isto be accepted.

According to an embodiment of the present invention, there is providedan information-processing method including the steps of:

reproducing content data common to the information-processing apparatusand the other information-processing apparatus synchronously with theother information-processing apparatus;

inputting first user condition information including a voice and imageof a user operating the information-processing apparatus;

receiving second user condition information including a voice and imageof an other user from the other information-processing apparatusoperated by the other user;

synthesizing a voice and image of the content data synchronouslyreproduced in a process carried out at the reproduction step with thevoice and image of the second user condition information received in aprocess carried out at the user-information receiver step;

analyzing at least either the first user condition information input ina process carried out at the user-information input step or the seconduser condition information received in a process carried out at theuser-information receiver step; and

setting a control parameter used for controlling a process, which iscarried out at the information synthesis step of synthesizing voices andimages, on the basis of an analysis result produced in a process carriedout at the information analysis step.

According to an embodiment of the present invention, there is provided arecording medium for recording a program as a recording medium. Theprogram includes the steps of:

reproducing content data common to the computer and theinformation-processing apparatus synchronously with theinformation-processing apparatus;

inputting first user condition information including a voice and imageof a user operating the computer;

receiving second user condition information including a voice and imageof an other user from the information-processing apparatus operated bythe other user;

synthesizing a voice and image of the content data synchronouslyreproduced in a process carried out at the reproduction step with thevoice and image of the second user condition information received in aprocess carried out at the user-information receiver step;

analyzing at least either the first user condition information input ina process carried out at the user-information input step or the seconduser condition information received in a process carried out at theuser-information receiver step; and

setting a control parameter used for controlling a process, which iscarried out at the information synthesis step of synthesizing voices andimages, on the basis of an analysis result produced in a process carriedout at the information analysis step.

According to an embodiment of the present invention, there is provided aprogram including the steps of:

reproducing content data common to the computer and theinformation-processing apparatus synchronously with theinformation-processing apparatus;

inputting first user condition information including a voice and imageof a user operating the computer;

receiving second user condition information including a voice and imageof an other user from the information-processing apparatus operated bythe other user;

synthesizing a voice and image of the content synchronously reproducedin a process carried out at the reproduction step with the voice andimage of the second user condition information received in a processcarried out at the user-information receiver step;

analyzing at least either the first user condition information input ina process carried out at the user-information input step or the seconduser condition information received in a process carried out at theuser-information receiver step; and

setting a control parameter used for controlling a process, which iscarried out at the information synthesis step of synthesizing voices andimages, on the basis of an analysis result produced in a process carriedout at the information analysis step.

According to an embodiment of the present invention, there is providedan information-processing apparatus including:

a reproduction section for reproducing content data common to theinformation-processing apparatus and the other information-processingapparatus synchronously with the other information-processing apparatus;

a user information input section for inputting first user conditioninformation including a voice and image of a user operating theinformation-processing apparatus;

a user information receiver section for receiving second user conditioninformation including a voice and image of an other user from the otherinformation-processing apparatus operated by the other user;

a synthesis section for synthesizing a voice and image of the contentdata synchronously reproduced by the reproduction section with the voiceand image of the second user condition information received by theuser-information receiver section;

an information analysis section for analyzing at least either the firstuser condition information input by the user-information input sectionor the second user condition information received by theuser-information receiver section; and

a parameter-setting section for setting a control parameter used forcontrolling a process, which is carried out by the synthesis section tosynthesize voices and images, on the basis of an analysis resultproduced by the information analysis section.

As described above, in this present invention, a content common to aninformation-processing apparatus and another information-processingapparatus is reproduced in the information-processing apparatussynchronously with the other information-processing apparatus. Then,first user condition information including a voice and image of a useroperating the information-processing apparatus is input. Subsequently,second user condition information including a voice and image of anotheruser is received from the other information-processing apparatusoperated by the other user. Then, a voice and image of the synchronouslyreproduced content are synthesized with the voice and image of thesecond user condition information. Subsequently, at least either theinput first user condition information or the received second usercondition information is analyzed. Finally, a control parameter used forcontrolling a process to synthesize voices and images is set on thebasis of the analysis result.

A network is a mechanism for connecting at least two apparatus to eachother and propagating information from one apparatus to another.Apparatus communicating with each other through the network can beindependent apparatus or internal blocks included in one apparatus.

Communication can of course be radio or wire communication. As analternative, communication can also be a combination of the radiocommunication and the wire communication, which are mixed with eachother. That is to say, the radio communication is adopted for certainareas while the wire communication is carried out for other areas. As analternative, the radio communication and the wire communication aremixed with each other by applying the radio communication tocommunications from a certain apparatus to another apparatus butapplying the wire communication to communications from the otherapparatus to the certain apparatus.

In accordance with an embodiment of the present invention, a synthesisof a plurality of images and a plurality of voices can be set with easein accordance with the conditions of users. In addition, in accordancewith an embodiment of the present invention, users present at locationsremote from each other are capable of communicating with each other in alively manner.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects of the invention will be seen by reference tothe description, taken in connection with the accompanying drawings, inwhich:

FIG. 1 is a diagram showing a typical configuration of a communicationsystem according to an embodiment of the present invention;

FIGS. 2A to 2C are diagrams showing a typical image of a content andtypical images of users in the communication system shown in FIG. 1;

FIGS. 3A to 3C are diagrams showing typical patterns of synthesis of acontent image with user images;

FIG. 4 is a block diagram showing a typical configuration of acommunication apparatus employed in the communication system shown inFIG. 1;

FIG. 5 shows a flowchart referred to in an explanation of remotecommunication processing carried out by the communication apparatusshown in FIG. 4;

FIG. 6 is a block diagram showing a detailed typical configuration of adata analysis section employed in the communication apparatus shown inFIG. 4;

FIG. 7 is a diagram referred to in explaining a typical method ofanalyzing information on conditions of users;

FIG. 8 is another diagram referred to in explaining the typical methodof analyzing the information on conditions of users;

FIG. 9 is a diagram referred to in explaining another typical method ofanalyzing the information on conditions of users;

FIG. 10 is another diagram referred to in explaining the other typicalmethod of analyzing information on conditions of users;

FIG. 11 is a diagram referred to in explaining a typical method ofanalyzing information on conditions of users by using a plurality ofpieces of real-time data;

FIG. 12 is another diagram referred to in explaining the typical methodof analyzing the information on conditions of users by using a pluralityof pieces of real-time data;

FIG. 13 shows a flowchart referred to in an explanation of auser-characteristic analysis mixing process carried out at a step S6 ofthe flowchart shown in FIG. 5;

FIG. 14 shows a flowchart referred to in an explanation of acontrol-information receiver process carried out as a counterpart of aprocess performed at a step S30 of the flowchart shown in FIG. 13; and

FIG. 15 is a block diagram showing a typical configuration of a personalcomputer according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before preferred embodiments of the present invention are explained,relations between disclosed inventions and the embodiments are explainedin the following comparative description. Even if there is an embodimentdescribed in this specification but not included in the followingcomparative description as an embodiment corresponding to an invention,such an embodiment is not to be interpreted as an embodiment notcorresponding to an invention. Conversely, an embodiment included in thefollowing comparative description as an embodiment corresponding to aspecific invention is not to be interpreted as an embodiment notcorresponding to an invention other than the specific invention.

In addition, the following comparative description is not to beinterpreted as a comprehensive description covering all inventionsdisclosed in this specification. In other words, the followingcomparative description by no means denies existence of inventionsdisclosed in this specification but not included in claims as inventionsfor which a patent application is filed. That is to say, the followingcomparative description by no means denies existence of inventions to beincluded in a separate application for a patent, included in anamendment to this specification, or added in the future.

An information-processing apparatus (such as a communication apparatus1-1 as shown in FIG. 1) according to an embodiment of the presentinvention includes:

reproduction means (such as a content reproduction section 25 as shownin FIG. 4) for reproducing content data common to thisinformation-processing apparatus and an other information-processingapparatus (such as a communication apparatus 1-2 shown in FIG. 1)synchronously with the other information-processing apparatus;

user-information input means (such as an input section 22-1 as shown inFIG. 4) for inputting first user condition information including a voiceand image of a user operating this information-processing apparatus;

user-information receiver means (such as a communication section 23 asshown in FIG. 4) for receiving second user condition informationincluding a voice and image of an other user from the otherinformation-processing apparatus operated by the other user;

synthesis means (such as an audio/video synthesis section 26 as shown inFIG. 4) for synthesizing a voice and image of the content synchronouslyreproduced by the reproduction means with the voice and image of thesecond user condition information received by the user-informationreceiver means;

information analysis means (such as a user-characteristic analysissection 71 as shown in FIG. 4) for analyzing at least either the firstuser condition information input by the user-information input means orthe second user condition information received by the user-informationreceiver means; and

parameter-setting means (such as a control-information generationsection 72 as shown in FIG. 4) for setting a control parameter used forcontrolling a process, which is carried out by the synthesis means tosynthesize voices and images, on the basis of an analysis resultproduced by the information analysis means.

It is also possible to provide a configuration in which theinformation-processing apparatus according to the embodiment of thepresent invention further includes analysis-object determination means(such as a synthesis control section 84 as shown in FIG. 4) forgenerating a determination result as to whether or not the second usercondition information received by the user-information receiver means isto be analyzed by the information analysis means and, if thedetermination result generated by the analysis-object determinationmeans indicates that the second user condition information is not to beanalyzed by the information analysis means, the information analysismeans analyzes only the first user condition information.

It is also possible to provide a configuration in which theinformation-processing apparatus according to the embodiment of thepresent invention further includes output means (such as an outputsection 21 as shown in FIG. 4) for outputting a voice and an image,which are obtained as a synthesis result carried out by the synthesismeans, and on the basis of an analysis result carried out by theinformation analysis means, the parameter-setting means also sets acontrol parameter of the output means.

It is also possible to provide a configuration in which theparameter-setting means sets a control parameter for controlling theother information-processing apparatus on the basis of an analysisresult carried out by the information analysis means and theinformation-processing apparatus according to the embodiment of thepresent invention further includes sender means (such as anoperation-information output section 87 as shown in FIG. 4) fortransmitting the control parameter set by the parameter-setting means tothe other information-processing apparatus.

It is also possible to provide a configuration in which theinformation-processing apparatus according to the embodiment of thepresent invention further includes acceptance determination means (suchas a session management section 81 as shown in FIG. 4) for generating adetermination result as to whether or not a control parametertransmitted by the other information-processing apparatus is to beaccepted.

An information-processing method according to an other embodiment of thepresent invention includes the steps of:

reproducing content data common to an information-processing apparatusand an other information-processing apparatus synchronously with theother information-processing apparatus (such as a step S4 of a flowchartshown in FIG. 5);

inputting first user condition information including a voice and imageof a user operating the information-processing apparatus (such as asub-step of carrying out a process in the input section 22-1 as shown inFIG. 4 at a step S2 of the flowchart shown in FIG. 5);

receiving second user condition information including a voice and imageof an other user from the other information-processing apparatusoperated by the other user (such as a sub-step of carrying out a processin the communication section 23 as shown in FIG. 4 at the step S2 of theflowchart shown in FIG. 5);

synthesizing a voice and image of the content data synchronouslyreproduced in a process carried out at the reproduction step with thevoice and image of the second user condition information received in aprocess carried out at the user-information receiver step (such as astep S28 of a flowchart shown in FIG. 13);

analyzing at least either the first user condition information input ina process carried out at the user-information input step or the seconduser condition information received in a process carried out at theuser-information receiver step (such as a step S23 of a flowchart shownin FIG. 13); and

setting a control parameter used for controlling a process, which iscarried out at the information synthesis step of synthesizing voices andimages, on the basis of an analysis result produced in a process carriedout at the information analysis step (such as a step S24 of a flowchartshown in FIG. 13).

It is to be noted that relations between a recording medium and aconcrete implementation according to an embodiment of the presentinvention are the same as the relations described above as relationsbetween the information-processing method and a concrete implementationin the embodiment. By the same token, relations between a program and aconcrete implementation in the embodiment are the same as the relationsdescribed above as relations between the information-processing methodand a concrete implementation in the embodiments. Thus, the relationsbetween the recording mediums and the concrete implementation in theembodiment as well as the relations between the program and the concreteimplementation in the embodiment are not explained to avoidduplications.

The embodiments of the present invention are explained in detail byreferring to drawings as follows.

FIG. 1 is a diagram showing a typical configuration of a communicationsystem according to an embodiment of the present invention. In thiscommunication system, a communication apparatus 1-1 is connected toanother communication apparatus 1 through a communication network 2. Inthe case of the typical configuration shown in FIG. 1, a communicationapparatus 1-2 serves as the other communication apparatus 1. Thecommunication apparatus 1-1 and 1-2 exchange images of their users aswell as user voices accompanying the images with each other in a waysimilar to the so-called television telephone. In addition, thecommunication apparatus 1-1 reproduces a content common to thecommunication apparatus 1-1 and 1-2 synchronously with the communicationapparatus 1-2. By displaying a common content in this way, remotecommunication between users is supported. In the following descriptions,the communication apparatus 1-1 and 1-2 are each referred to simply asthe communication apparatus 1 in case it is not necessary to distinguishthe communication apparatus 1-1 and 1-2 from each other.

It is to be noted that examples of the common content are a programcontent obtained as a result of receiving a television broadcast, thecontent of an already acquired movie or the like obtained bydownloading, a private content exchanged between users, a game content,a musical content and a content prerecorded on an optical diskrepresented by a DVD (Digital Versatile Disk). It is to be noted thatthe optical disk itself is not shown in the figure.

The communication apparatus 1 can be utilized by a plurality of users atthe same time. In the case of the typical configuration shown in FIG. 1,for example, the communication apparatus 1-1 is utilized by users A andB whereas the communication apparatus 1-2 is utilized by a user X.

As an example, an image of a common content is shown in FIG. 2A. Animage taken by the communication apparatus 1-1 is an image of the user Alike one shown in FIG. 2B. On the other hand, an image taken by thecommunication apparatus 1-2 is an image of the user X like one shown inFIG. 2C. In this case, a display unit 41 employed in the communicationapparatus 1-1 as shown in FIG. 4 displays a picture-in-picture screenlike one shown in FIG. 3A, a cross-fade screen like one shown in FIG.3B, or a wipe screen like one shown in FIG. 3C. In either case, theimage of the common content and the images of the users are superposedon each other.

On the picture-in-picture display like the one shown in FIG. 3A, theimages of the users are each superposed on the image of the commoncontent as a small screen. The position and size of each of the smallscreens can be changed in an arbitrary manner. In addition, instead ofdisplaying the images of both the users, that is, instead of displayingthe image of the user A itself and the image of the user X serving as acommunication partner of the user A, only the image of either of theusers can be displayed.

In the cross-fade screen like the one shown in FIG. 3B, the image of thecommon content is synthesized with the image of a user, which can be theuser A or X. This cross-fade screen can be used for example when theuser points to an arbitrary position or area on the image of the commoncontent.

In the wipe screen like the one shown in FIG. 3C, the image of a userappears on the screen while moving in a certain direction, graduallycovering the image of the common content. In the typical screen shown inFIG. 3C, the image of the user appears from the right side.

The above synthesis patterns of the screen can be changed from time totime. In addition, each of the synthesis patterns has synthesisparameters such as image balance to set the transparency of each imagein the synthesis patterns shown in FIGS. 3A to 3C and volume balance toset the volumes of the content and the users. These synthesis parameterscan also be changed from time to time. A history showing changes of thesynthesis pattern from one to another and changes of the synthesisparameters is stored in a synthesis-information storage section 64 asshown in FIG. 4. It is to be noted that the pattern to display the imageof the content and the images of the users is not limited to thesynthesis patterns described above. That is to say, the images can alsobe displayed as a synthesis pattern other than the patterns describedabove.

Refer back to FIG. 1. The communication network 2 is a broad-band datacommunication network represented by typically the Internet. At arequest made by the communication apparatus 1, a content-providingserver 3 supplies a content to the communication apparatus 1 by way ofthe communication network 2. Before the user of the communicationapparatus 1 can utilize the communication system, an authenticationserver 4 authenticates the user. In addition, the authentication server4 also carries out an accounting process and other processing for asuccessfully authenticated user.

A broadcasting apparatus 5 is a unit for transmitting a content, whichis typically a program of a television broadcast or the like. Thus, thecommunication apparatus 1 are capable of receiving and reproducing thecontent from the broadcasting apparatus 5 in a synchronous manner. It isto be noted that the broadcasting apparatus 5 is capable of transmittinga content to the communication apparatus 1 by radio or wirecommunication. In addition, the broadcasting apparatus 5 may alsotransmit a content to the communication apparatus 1 by way of thecommunication network 2.

A standard-time information broadcasting apparatus 6 is a unit forsupplying information on a standard time to the communication apparatus1. The standard time information is used for correctly synchronizing astandard-time measurement section 30, which is employed in each of thecommunication apparatus 1 as shown in FIG. 4 to serve as a clock, to astandard time. The standard time measured by a clock can typically bethe world or Japanese standard time. It is to be noted that thestandard-time information broadcasting apparatus 6 is capable oftransmitting the information on a standard time to the communicationapparatus 1 by radio or wire communication. In addition, thestandard-time information broadcasting apparatus 6 may also transmit theinformation on a standard time to the communication apparatus 1 by wayof the communication network 2.

In the typical communication system shown in FIG. 1, only twocommunication apparatus 1 are connected to each other by thecommunication network 2. It is also worth noting, however, that thenumber of communication apparatus 1 connected to the communicationnetwork 2 is not limited to two. That is to say, any plurality ofcommunication apparatus 1 including communication apparatus 1-3 and 1-4can be connected to each other by the communication network 2.

Next, a typical configuration of the communication apparatus 1-1 isexplained in detail by referring to FIG. 4.

An output section 21 employed in the communication apparatus 1-1includes a display unit 41 and a speaker 42. The output section 21displays an image corresponding to a video signal received from anaudio/video synthesis section 26 on the display unit 41 and outputs asound corresponding to an audio signal received from the audio/videosynthesis section 26 to the speaker 42.

The input section 22-1 includes a camera 51-1, a microphone 52-1, and asensor 53-1. By the same token, the input section 22-2 includes a camera51-2, a microphone 52-2, and a sensor 53-2. In the followingdescriptions, the input sections 22-1 and 22-2 are each referred tosimply as the input section 22 in case it is not necessary todistinguish the input sections 22-1 and 22-2 from each other. In thesame way, the cameras 51-1 and 51-2 are each referred to simply as thecamera 51 in case it is not necessary to distinguish the cameras 51-2and 51-2 from each other. By the same token, the microphones 52-1 and52-2 are each referred to simply as the microphone 52 in case it is notnecessary to distinguish the microphones 52-1 and 52-2 from each other.Likewise, the sensors 53-1 and 53-2 are each referred to simply as thesensor 53 in case it is not necessary to distinguish the sensors 53-1and 53-2 from each other.

The camera 51 is a component for taking an image of the user. The imageof the user can be a moving or still image. The microphone 52 is acomponent for collecting voices of the user and other sounds. The sensor53 is a component for detecting information on an environmentsurrounding the user. The information on the environment includes thebrightness, the ambient temperature, and the humidity.

The input section 22 outputs the acquired image, voices/sounds, andinformation on the environment to a communication section 23, a storagesection 27, and a data analysis section 28 as RT (Real Time) data of theuser. In addition, the input section 22 also outputs the acquired userimage and user voices to the audio/video synthesis section 26.

It is to be noted that a plurality of input sections 22 can also beprovided, being oriented toward a plurality of respective users. In thecase of the communication apparatus 1-1 shown in FIG. 4, for example,two input sections 22 are provided, being oriented toward the two usersA and B shown in FIG. 1.

The communication section 23 is a unit for transmitting real-time datainput by the input section 22 as data of the users A and/or B to thecommunication apparatus 1-2 serving as a communication partner by way ofthe communication network 2 and receiving real-time data of the user Xfrom the communication apparatus 1-2. The communication section 23supplies the real-time data of the user X to the audio/video synthesissection 26 and the storage section 27. In addition, the communicationsection 23 also receives a content transmitted by the communicationapparatus 1-2 or the content-providing server 3 by way of thecommunication network 2 and supplies the content to a contentreproduction section 25 and the storage section 27. Such a content isalso referred to hereafter as content data. The communication section 23transmits a content and information to the communication apparatus 1-2by way of the communication network 2. The content is a content read outfrom the storage section 27 and the information is operation informationand control information generated by an operation-information outputsection 87.

A broadcast receiver section 24 is a unit for receiving a televisionbroadcast signal broadcasted by the broadcasting apparatus 5 andsupplying a broadcasted program conveyed by the signal as a content tothe content reproduction section 25 and, if necessary, to the storagesection 37. The content reproduction section 25 is a unit forreproducing a content, which is a broadcasted program received by thebroadcast receiver section 24. The reproduced content may also a contentreceived by the communication section 23, a content read out from thestorage section 27, or a content read out from a disk such as an opticaldisk. It is to be noted that the disk itself is not shown in the figure.The content reproduction section 25 supplies a sound and image of thereproduced content to the audio/video synthesis section 26 and the imageanalysis section 28.

The audio/video synthesis section 26 is a unit for mixing an image andsound received from the content reproduction section 25 as an image andsound of a content, an image and voice received from the input section22 as an image and voice of the user A, an image and voice received fromthe communication section 23 as an image and voice of the user X as wellas a character string for typically arousing the alert for the user Aand supplying a video signal obtained as a analysis result to the outputsection 21. Referred to hereafter as a synthesis process, the mixingprocess carried out by the audio/video synthesis section 26 is a processof blending and adjusting images, sounds, voices, and character strings.

The storage section 27 includes a content storage section 61, a licensestorage section 62, a user-information storage section 63, and thesynthesis-information storage section 64 mentioned before. The contentstorage section 61 is a unit for storing data received from the inputsection 22 as real-time data of a user such as the user A, data receivedfrom the communication section 23 as real-time data of the communicationpartner such as the user X, a broadcast program received from thebroadcast receiver section 24 as a content, and a content received fromthe communication section 23. The license storage section 62 is a unitfor storing information such as a license granted to the communicationapparatus 1-1 as a license for utilizing a content stored in the contentstorage section 61. The user-information storage section 63 is a unitfor storing data such as information on privacy of a group to which thecommunication apparatus 1-1 pertains. The synthesis-information storagesection 64 is a unit for storing each synthesis pattern and everysynthesis parameter, which can be changed by a synthesis control section84.

Composed of a user-characteristic analysis section 71 and acontrol-information generation section 72, the data analysis section 28is a unit for inputting data received from the input section 22 asreal-time data of a user such as the user A, data received from thecommunication section 23 as real-time data of the communication partnersuch as the user X, and a content received from the content reproductionsection 25.

The user-characteristic analysis section 71 is a unit for analyzingcharacteristics of information including the posture of each user, thenumber of users, voices of each user, and the condition of users andsupplying the analysis result to the control-information generationsection 72. The condition of users is information on an environmentsurrounding each of the users. The information, the characteristics ofwhich are analyzed, is information obtained from data received from theinput section 22 as real-time data of the users including the user A. Inaddition to the characteristics of the information, theuser-characteristic analysis section 71 also analyzes characteristics ofother information and supplying the analysis result to thecontrol-information generation section 72. The other information isinformation obtained from data received from the communication section23 as real-time data of a user such as the user X.

The control-information generation section 72 is a unit for generatingcontrol information used for executing control of sections employed inthe communication apparatus 1-1 in accordance with the analysis resultsreceived from the user-characteristic analysis section 71 and outputtingthe control information to the control section 32. For example, thecontrol-information generation section 72 generates control informationto be used for executing control of the audio/video synthesis section 26and outputs the control information to the control section 32. In thiscase, the audio/video synthesis section 26 is controlled to synthesizean image and voice of data received from the communication section 23 asreal-time data of the communication partner such as the user X and animage and sound of a content received from the content reproductionsection 25 on the basis of synthesis parameters and synthesis patternaccording to the analysis result received from the user-characteristicanalysis section 71.

In addition, the control-information generation section 72 alsogenerates control information to be used for executing control ofsections employed in the communication apparatus 1-2, which serves asthe communication partner, in accordance with the analysis resultsreceived from the user-characteristic analysis section 71 and outputsthe control information to the control section 32.

A communication-environment detection section 29 is a unit formonitoring an environment of communication with the communicationapparatus 1-2 through the communication section 23 and the communicationnetwork 2 and outputting a result of the monitoring to the controlsection 32. The environment of communication includes a communicationrate and a communication delay time. A standard-time measurement section30 is a unit for adjusting a standard time measured by itself on thebasis of a standard time received from the standard-time informationbroadcasting apparatus 6 and supplying the adjusted standard time to thecontrol section 32. An operation input section 31 is typically a remotecontroller for accepting an operation carried out by the user andissuing a command corresponding to the operation to the control section32.

The control section 32 is a unit for controlling other components of thecommunication apparatus 1-1 on the basis of information such as a signalrepresenting an operation received by the operation input section 31 asan operation carried out by the user and control information receivedfrom the data analysis section 28. The control section 32 includes asession management section 81, a viewing/listening recording levelsetting section 82, a reproduction synchronization section 83, theaforementioned synthesis control section 84, a reproduction permissionsection 85, a recording permission section 86, the operation-informationoutput section 87 mentioned above, and an electronic-apparatus controlsection 88. It is to be noted that, in the typical configuration shownin FIG. 4, control lines used for outputting control commands from thecontrol section 32 to other components of the communication apparatus1-1 are omitted.

The session management section 81 is a unit for controlling a processcarried out by the communication section 23 to connect the communicationapparatus 1-1 to other apparatus such as the communication apparatus1-2, the content-providing server 3, and the authentication server 4through the communication network 2. In addition, the session managementsection 81 also determines whether or not to accept control informationreceived from another apparatus such as the communication apparatus 1-2as information used for controlling sections employed in thecommunication apparatus 1-1.

The viewing/listening recording level setting section 82 is a unit fordetermining whether or not real-time data acquired by the input section22 and/or a content stored in the content storage section 61 as apersonal content of the user can be reproduced and recorded by thecommunication apparatus 1-2, which serves as the communication partner,on the basis of an operation carried out by the user. If the real-timedata and/or the personal content are determined to be data and/or acontent that can be recorded by the communication apparatus 1-2, themaximum number of times the data and/or the content can be recorded andother information are set. This set information is added to thereal-time data of the user as privacy information and transmitted to thecommunication apparatus 1-2 from the communication section 23. Thereproduction synchronization section 83 is a unit for controlling thecontent reproduction section 25 to reproduce a content common to thecommunication apparatus 1-2 synchronously with the communicationapparatus 1-2, which serves as the communication partner.

The synthesis control section 84 is a unit for controlling the dataanalysis section 28 to analyze user-condition information, which isobtained from real-time data of the user, on the basis of an operationcarried out by the user. In addition, the synthesis control section 84also controls the audio/video synthesis section 26 to synthesize animage of a content with images of users and synthesize a voice of acontent with voices of users in accordance with an operation carried outby the user or control information received from the data analysissection 28. That is to say, on the basis of the control informationreceived from the data analysis section 28, the synthesis controlsection 84 changes setting of the synthesis pattern to any of thepatterns shown in FIGS. 3A to 3C and setting of synthesis parameters ofthe newly set synthesis pattern. The synthesis control section 84 thencontrols the audio/video synthesis section 26 in accordance with thenewly set synthesis pattern and synthesis parameters. In addition, thesynthesis control section 84 records the newly set synthesis pattern andsynthesis parameters in the synthesis-information storage section 64 assynthesis information.

The reproduction permission section 85 is a unit for outputting adetermination result as to whether or not a content can be reproduced onthe basis of information such as a license attached to the contentand/or the privacy information set by the viewing/listening recordinglevel setting section 82 employed in the communication partner andcontrolling the content reproduction section 25 on the basis of thedetermination result. The recording permission section 86 is a unit foroutputting a determination result as to whether or not a content can berecorded on the basis of information including a license attached to thecontent and/or the privacy information and controlling the storagesection 27 on the basis of the determination result.

The operation-information output section 87 is a unit for generatingoperation information for an operation carried out by the user andtransmitting the information to the communication apparatus 1-2 servingas the communication partner by way of the communication section 23. Theoperation carried out by the user can be an operation to change achannel to receive a television broadcast, an operation to start aprocess to reproduce a content, an operation to end a process toreproduce a content, an operation to reproduce a content in afast-forward process, or another operation. The operation informationincludes a description of the operation and a time at which theoperation is carried out. Details of the operation information will bedescribed later. The operation information is used in synchronousreproduction of a content. In addition, the operation-information outputsection 87 also transmits control information received from the dataanalysis section 28 to the communication apparatus 1-2 by way of thecommunication section 23.

The electronic-apparatus control section 88 is a unit for setting theoutput of the output section 21, setting the input of the input section22, and controlling a predetermined electronic apparatus, which isconnected to the communication apparatus 1-1 as a peripheral, on thebasis of an operation carried out by the user or control informationreceived from the data analysis section 28. Examples of thepredetermined electronic apparatus are an illumination apparatus and anair-conditioning apparatus, which are not shown in the figure.

It is to be noted that, since a detailed typical configuration of thecommunication apparatus 1-2 is the same as that of the communicationapparatus 1-1 shown in FIG. 4, no special explanation of the detailedtypical configuration of the communication apparatus 1-2 is given.

Next, remote communication processing carried out by the communicationapparatus 1-1 to communicate with the communication apparatus 1-2 isexplained by referring to a flowchart shown in FIG. 5 as follows. It isto be noted that the communication apparatus 1-2 also carries out thisprocessing in the same way as the communication apparatus 1-1.

The remote communication processing to communicate with thecommunication apparatus 1-2 is started when an operation to start theremote communication is carried out by the user on the operation inputsection 31 and an operation signal corresponding to the operation issupplied by the operation input section 31 to the control section 32.

The flowchart shown in the figure begins with a step S1 at which thecommunication section 23 establishes a connection with the communicationapparatus 1-2 through the communication network 2 on the basis ofcontrol executed by the session management section 81 in order to notifythe communication apparatus 1-2 that a remote communication is started.Then, the flow of the processing goes on to a step S2. In response tothis notification, the communication apparatus 1-2 returns anacknowledgement of the notification to the communication apparatus 1-1as an acceptance of the start of the remote communication.

At the step S2, the communication section 23 starts transmittingreal-time data of the user A and other real-time data, which arereceived from the input section 22, by way of the communication network2 on the basis of control executed by the control section 32. Thecommunication section 23 also starts receiving real-time data of theuser X from the communication apparatus 1-2. Then, the flow of theprocessing goes on to a step S3. At that time, data received from theinput section 22 as the real-time data of the user A and the otherreal-time data as well as real-time data received from the communicationapparatus 1-2 as the real-time data of the user X are supplied to thedata analysis section 28. An image and voice included in the real-timedata of the user A and an image and voice included the other real-timedata as well as an image and voice included in the real-time data of theuser X are supplied to the audio/video synthesis section 26.

At the step S3, the communication section 23 establishes a connectionwith the authentication server 4 through the communication network 2 onthe basis of control, which is executed by the session managementsection 81, in order to carry out an authentication process foracquiring a content. After the authentication process has been completedsuccessfully, the communication section 23 makes an access to thecontent-providing server 3 through the communication network 2 in orderto acquire a content specified by the user. Then, the flow of theprocessing goes on to a step S4. In the mean time, the communicationapparatus 1-2 carries out the same processes as the communicationapparatus 1-1 to obtain the same content.

It is to be noted that, if the specified content is a content to bereceived as a television broadcast or an already acquired content storedin the storage section 27 and ready for reproduction, the process of thestep S3 can be omitted.

At the step S4, the content reproduction section 25 starts a process toreproduce the content synchronized with the communication apparatus 1-2on the basis of control executed by the reproduction synchronizationsection 83. Then, the flow of the processing goes on to a step S5. Bycarrying out the process to reproduce the content synchronized with thecommunication apparatus 1-2, the communication apparatus 1-1 and 1-2reproduce the same content in a synchronous manner on the basis of astandard time supplied by the standard-time measurement section 30 (orthe standard-time information broadcasting apparatus 6). The reproducedcontent is supplied to the audio/video synthesis section 26 and the datasynthesis section 28.

At the step S5, the storage section 27 starts a remote communicationrecording process. Then, the flow of the processing goes on to a stepS6. To put it concretely, the audio/video synthesis section 26synthesizes the content, the reproduction of which has been started, theimages and voices included in the input real-time data of the user A,and the other input real-time data as well as the image and voicesincluded in the received real-time data of the user X in accordance withcontrol executed by the synthesis control section 84. Then, theaudio/video synthesis section 26 supplies audio and video signalsobtained as a analysis result process to the output section 21. It is tobe noted that, at that time, the synthesis control section 84 controlsthe synthesis process, which is carried out by the audio/video synthesissection 26, on the basis of a synthesis pattern and synthesis parametersfor the pattern. As described earlier, the synthesis pattern andsynthesis parameters for the pattern have been set in advance inaccordance with an operation carried out by the user.

The output section 21 displays an image based on the video signalsupplied thereto and generates a sound based on the received audiosignal. At this stage, exchanges of an image and a voice between theusers and a process to reproduce a content in a synchronous manner havebeen started.

Then, the start of the exchanges of an image and a voice between theusers and the process to reproduce a content in a synchronous manner isfollowed by a start of a process to record the content, the reproductionof which has been started, the images and voices included in thereal-time data of the user A and the other real-time data as well as theimages and voices included in the real-time data of the user X, andsynthesis information including the synthesis pattern and the synthesisparameters set for the synthesis pattern.

At the step S6, in accordance with control executed by the synthesiscontrol section 84, the data analysis section 28 and the audio/videosynthesis section 26 carry out a user-characteristic analysis mixingprocess, details of which will be described later. To be more specific,at the step S6, the data analysis section 28 analyzes information on theconditions of users. Obtained from real-time data of each user, theinformation on the conditions of users includes information on theposture of each user, the number of users, a voice of each user, andinformation on the environment surrounding each user. Then, the dataanalysis section 28 generates control information, which is used forcontrolling sections including the audio/video synthesis section 26, onthe basis of the analysis result. In this way, the synthesis controlsection 84 carries out a process to control synthesis processingexecuted by the audio/video synthesis section 26 by changing thesynthesis pattern to another and properly setting synthesis parametersof the new synthesis pattern on the basis of the control information inplace of a synthesis pattern determined in advance in accordance with anoperation performed by the user and synthesis parameters set in advancefor the determined synthesis pattern.

Then, at the next step S7, the control section 32 produces adetermination result as to whether or not the user has carried out anoperation to make a request for termination of the remote communication.The control section 32 carries out the process of this step repeatedlyuntil the user carries out such an operation. As the determinationresult produced at the step S7 indicates that the user has carried outan operation to make a request for termination of the remotecommunication, the flow of the processing goes on to a step S8.

At the step S8, the communication section 23 establishes a connectionwith the communication apparatus 1-2 through the communication network 2on the basis of control, which is executed by the session managementsection 81, in order to notify the communication apparatus 1-2 that aremote communication has been ended. In response to this notice, thecommunication apparatus 1-2 returns an acknowledgement of thenotification to the communication apparatus 1-1 as an acceptance of thetermination of the remote communication.

Then, at the next step S9, the storage section 27 terminates theremote-communication-recording process. It is to be noted that, in thisway, when a next remote communication is carried out later on, it ispossible to utilize the stored data of the terminated remotecommunication. The stored data of the terminated remote communicationincludes the reproduced content, the images and voices included in thereal-time data of the user A and the other real-time data as well as theimages and voices included in the real-time data of the user X, and thesynthesis information described above.

The processing of the remote communication processing carried out by thecommunication apparatus 1-1 as communication processing between thecommunication apparatus 1-1 and the communication apparatus 1-2 has beenexplained above.

The following description explains details of the aforementioneduser-characteristic analysis mixing process carried out at the step S6of the flowchart representing the remote communication processingdescribed above.

FIG. 6 is a block diagram showing a detailed configuration of the dataanalysis section 28 for carrying out the user-characteristic analysismixing process. It is to be noted that, specific configuration sectionsshown in FIG. 6 as sections identical with their respective counterpartsemployed in the configuration shown in FIG. 4 are denoted by the samereference numerals as the counterparts and description of the specificconfiguration sections is omitted to avoid duplications.

As shown in FIG. 6, a typical configuration of the user-characteristicanalysis section 71 includes an analysis control section 101, aposture-information analysis section 102, a user-count analysis section103, an audio-information analysis section 104 and anenvironment-information analysis section 105.

The analysis control section 101 is a unit for controlling sections,which are provided for analyzing information, in accordance with controlexecuted by the synthesis control section 84, to analyze conditioninformation (or characteristics) of users and supplying the analysisresult to the control-information generation section 72. The sectionsprovided for analyzing information are the posture-information analysissection 102, the user-count analysis section 103, the audio-informationanalysis section 104 and the environment-information analysis section105. The condition information (or characteristics) of users isinformation obtained from real-time data of the users as real-time datato be analyzed. The real-time data to be analyzed is data extracted frominput real-time data of the user A and received real-time data of theuser X.

The posture-information analysis section 102 is a unit for carrying outan analysis process based on information on front, rear, right andleft-leaning postures of the user and supplying the analysis resultprocess to the analysis control section 101. The information on forward,backward, right and left-leaning postures of the user is informationextracted from an image included in the real-time data of the user asreal-time data to be analyzed. The user-count analysis section 103 is aunit for analyzing the number of users, who participate in the remotecommunication, on the basis of the input real-time data, and supplyingthe analysis result to the analysis control section 101. Theaudio-information analysis section 104 is a unit for carrying out ananalysis process based on audio information extracted from soundsincluded in the real-time data to be analyzed and supplying the analysisresult to the analysis control section 101. Examples of the audioinformation are the volume and frequency of a sound.

The environment-information analysis section 105 is a unit for carryingout an analysis process based on information on an environmentsurrounding the user and supplying the analysis result to the analysiscontrol section 101. Extracted from the real-time data to be analyzed,the information on an environment surrounding the user includesbrightness and an ambient temperature. It is to be noted that, inaddition to the brightness and the ambient temperature, the informationon an environment surrounding the user may also include humidity, anacceleration of a movement of the communication apparatus 1-1, theposition of the communication apparatus 1-1 and a time of the movement.In this case, the input section 22 may be provided with a sensor foreach item of the input information on an environment surrounding theuser. As an alternative, the communication apparatus 1-1 is providedwith an internal GIS (Geographical Information System) function.

On the basis of analysis results produced in accordance with controlexecuted by the analysis control section 101, the control-informationgeneration section 72 generates control information to be used forcontrolling processes carried out by sections employed in thecommunication apparatus 1-1. The control-information generation section72 then supplies the control information to the synthesis controlsection 84 or the electronic-apparatus control section 88. In addition,also on the basis of analysis results received from the analysis controlsection 101, the control-information generation section 72 generatescontrol information to be used for controlling processes carried out bysections employed in the communication apparatus 1-2. In this case, thecontrol-information generation section 72 supplies the controlinformation to the operation-information output section 87.

By referring to FIGS. 7 and 8, the following description concretelyexplains a method to analyze information on the condition of a user byusing real-time data in the user-characteristic analysis mixing process.It is to be noted that examples of FIGS. 7 and 8 show processing carriedout by the control-information generation section 72 to generate controlinformation on the basis of an analysis result performed by theposture-information analysis section 102.

In the example shown in FIG. 7, the user A utilizes the communicationapparatus 1-1 to adopt the picture-in-picture method explained earlierby referring to FIG. 3A as a method for carrying out the remotecommunication recording process at the step S5 of the flowchart shown inFIG. 5 to communicate with the user X operating the communicationapparatus 1-2.

Thus, the input section 22 employed in the communication apparatus 1-1is used to acquire real-time data including an image and voice of theuser A, brightness as well as an ambient temperature and transmit thedata to the communication apparatus 1-2 by way of the communicationsection 23. On the other hand, the communication apparatus 1-2 transmitsthe real-time data of the user X to the communication section 23employed in the communication apparatus 1-1. It is to be noted that FIG.7 shows the camera 51 as the input section 22.

Then, the display unit 41 employed in the communication apparatus 1-1displays an image of the user X serving as the communication partner ona child screen 201 superposed on an image 202 of a content beingreproduced at the right bottom corner of the full screen. It is to benoted that the transparency of the child screen 201 is set at a standardtransparency prior to the start of the user-characteristic analysismixing process.

At that time, the user A carries out an operation to start theuser-characteristic analysis mixing process by using the operation inputsection 31.

The user-characteristic analysis section 71 has received real-time dataof the users A and X. The posture-information analysis section 102employed in the user-characteristic analysis section 71 extracts contourinformation G1 from an image 211 taken by the camera 51 as shown in FIG.8 as an image included in the real-time data of the user A. Then, theposture-information analysis section 102 analyzes the extracted contourinformation G1 to determine whether the forward/backward-leaning postureinformation (or information on characteristics of the posture) of theuser A is information indicating a forward or backward-leaning postureby comparison of the extracted contour information G1 with a referencecontour G0 of the user A. That is to say, the posture-informationanalysis section 102 analyzes the extracted contour information G1 todetermine whether the posture information of the user A is a forward orbackward-leaning posture, which is displayed by the user A as a posturewith respect to the display unit 41. In this way, theposture-information analysis section 102 is capable of recognizing thelevel of enthusiasm expressed by the user A as an enthusiasm for thecontent being reproduced. The forward/backward-leaning posture displayedby the user A is a posture inclining in the forward or backwarddirection.

Then, the control-information generation section 72 generates controlinformation, which will be used for controlling the degree oftransparency of the child screen 201 displaying the image of the user Xin the synthesis of images, on the basis of a result of a processcarried out by the posture-information analysis section 102 to analyzethe information on the posture. In addition, the control-informationgeneration section 72 also generates control information, which will beused for controlling the balance between the volumes of the voices ofthe users A and X in the synthesis of sounds, on the basis of a resultof a process carried out by the posture-information analysis section 102to analyze the information on the posture.

The reference contour G0 cited above is set in advance by theposture-information analysis section 102. Typically, the referencecontour G0 is set as follows. First of all, images of the user A havinga reference posture are taken by using the camera 51. The referenceposture is a posture in which the reference contour G0 is set. Fromimages taken by the camera 51, the reference posture of the user A isthen determined by adoption of a method such as a motion vectoranalysis. Finally, information on the reference contour G0 of thereference posture of the user A is acquired. It is to be noted that, asan alternative to the adoption of the motion vector analysis, thereference contour G0 can also be set by letting the user A confirm theuser itself appearing on the display unit 41.

Then, the posture-information analysis section 102 finds a contour ratioe (=G1/G0) where symbol G0 denotes the reference-contour information ofthe user A and symbol G1 denotes the current-contour information of theuser A. Subsequently, the posture-information analysis section 102analyzes the computed contour e to determine whether the posture of theuser A is a forward-leaning or backward-leaning posture on the basis ofa graph 212 shown in FIG. 8 as follows.

The vertical axis of the right-side diagram showing the graph 212 inFIG. 8 represents information on the posture of the user A. The centerof the vertical axis is the posture information corresponding to thereference posture. Posture information above the center representsforward-leaning postures and posture information below the centerrepresents backward-leaning postures. On the other hand, the horizontalaxis represents the contour ratio e. The center of the horizontal axisis the contour ratio e of 1. Contour ratios e on the right side of thecenter are contour ratios e greater than 1, each representing a positionon the +side corresponding to forward-leaning postures. On the otherhand, contour ratios e on the left side of the center are contour ratiose smaller than 1, each representing a position on the − sidecorresponding to backward-leaning postures.

Thus, for a contour ratio e smaller than 1, that is, for a currentcontour G1 of the user A smaller than the reference contour G0, theposture-information analysis section 102 determines that the posture ofthe user A is leaning backward in comparison with the reference posture,generating an analysis result indicating that the user A has a low levelof enthusiasm for the content being reproduced. In this case, on thebasis of such the analysis result, as shown by a child screen 201Bdisplaying the image of the user X, the control-information generationsection 72 lowers the degree of transparency of the child screen 201B toa value smaller than the degree of transparency of the reference childscreen 201 cited earlier. That is to say, the posture-informationanalysis section 102 generates control information to raise theconcentration of the image of the user X and increase the volume of avoice of the user X to let the user better view and listen to the userX.

If the contour ratio e is found greater than 1 or the current contour G1of the user A is found greater than the reference contour G0 in the dataanalysis section 28, the posture-information analysis section 102determines that the posture of the user A is leaning forward incomparison with the reference posture, producing an analysis resultindicating that the user A has a high level of enthusiasm for thecontent being reproduced. In this case, on the basis of such theanalysis result, as shown by a child screen 201F displaying the image ofthe user X, the control-information generation section 72 raises thedegree of transparency of the child screen 201F to a value greater thanthe degree of transparency of the reference child screen 201 citedearlier. That is to say, the posture-information analysis section 102generates control information to decrease the concentration of the imageof the user X and reduce the volume of a voice of the user X to let theuser better view and listen to the content.

In the case of the example shown in FIG. 7, the degree of transparencyof the child screen 201 displaying the image of a communication partneris adjusted. It is to be noted, however, that instead of adjusting thedegree of transparency, the size of the child screen 201 can also beadjusted.

As described above, in the case of a forward-leaning posture of the userA, the synthesis is carried out to produce a display with the image ofthe user X not striking. Thus, the user A is capable of easily obtainingan environment allowing a content to be viewed and listened to with easewithout setting the display of the child screen by the user itself. Inthe case of the backward-leaning posture of the user A, on the otherhand, the synthesis is carried out to produce a display with the imageof the user X striking. Thus, the user A is capable of easily obtainingan environment allowing a communication with the user X to be carriedout with ease.

In the case of the example shown in FIG. 8, a contour extracted from animage of the user A is analyzed to determine whether the posture of theuser A is a forward or backward-leaning posture. It is to be noted,however, that the analysis method is not limited to this technique. Forexample, instead of extracting a counter from an image, it is possibleto provide a configuration in which 2 cameras 51 are provided on theright and left sides of the display unit 41 and information on a postureis extracted by using a stereo view generated from images taken by thecameras 51 to be used in the analysis of the posture. As anotheralternative, it is possible to provide a configuration in which aninfrared-ray sensor is installed at a location close to the display unit41 and a 3-dimensional distance measurement method using an infrared rayis adopted to extract posture information to be analyzed.

Next, by referring to FIGS. 9 and 10, another typical analysis method ofanalyzing information on the condition of a user by using the real-timedata of the user is explained in concrete terms. It is to be noted thatexamples shown in FIGS. 9 and 10 are other examples of the processingexplained earlier by referring to FIGS. 7 and 8. In the case of theexamples shown in FIGS. 9 and 10, however, there are 2 communicationpartners, i.e., the users X and Y. Also in the case of the examplesshown in FIGS. 9 and 10, the control-information generation section 72generates control information on the basis of an analysis result carriedout by the posture-information analysis section 102.

In the case of the example shown in FIG. 9, the user A utilizes thecommunication apparatus 1-1 to adopt the picture-in-picture methodexplained earlier by referring to FIG. 3A as a method for carrying outthe remote communication recording process at the step S5 of theflowchart shown in FIG. 5 to communicate with the user X operating thecommunication apparatus 1-2 and a user Y operating another communicationapparatus 1-3 shown in none of the figures.

Thus, the input section 22 employed in the communication apparatus 1-1is used to acquire real-time data including an image and voice of theuser A, brightness and an ambient temperature and transmit the data tothe communication apparatus 1-2 and 1-3 by way of the communicationsection 23. On the other hand, the communication apparatus 1-2 and 1-3transmit the real-time data of the users X and Y to the communicationsection 23 employed in the communication apparatus 1-1. It is to benoted that FIG. 9 shows the camera 51 as the input section 22.

Then, the display unit 41 employed in the communication apparatus 1-1displays an image of the user X serving as a communication partner on achild screen 221 superposed on an image 223 of a content beingreproduced on the right side on the image 223. At the same time, thedisplay unit 41 also displays an image of the user Y serving as anothercommunication partner on a child screen 222 superposed on the image 223on the left side of the image 223. It is to be noted that the sizes ofthe child screens 221 and 222 are each set at a standard size prior tothe start of the user-characteristic analysis mixing process.

At that time, the user A carries out an operation to start theuser-characteristic analysis mixing process by using the operation inputsection 31.

The user-characteristic analysis section 71 has received real-time dataof the users A, X and Y. The posture-information analysis section 102employed in the user-characteristic analysis section 71 extracts acurrent head position H1 from an image 231 taken by the camera 51 asshown in FIG. 10 as an image included in the real-time data of the userA. Then, the posture-information analysis section 102 analyzes theextracted current head position H1 by comparison of the head position H1of the user A with a reference head position H0 of the user A in orderto determine whether the posture of the user A is right or left leaning.That is to say, the posture-information analysis section 102 determineswhether the posture of the user A is right or left leaning with respectto the center of the screen of the display unit 41. In other words, theposture-information analysis section 102 determines whether the postureof the user A is a posture oriented in a direction toward the childscreen 221 or 222. The left/right leaning posture displayed by the userA is a posture inclining in the left or right direction.

Then, the control-information generation section 72 generates controlinformation, which will be used for controlling the sizes the childscreens 221 and 222 in the synthesis of images, on the basis of a resultof a process carried out by the posture-information analysis section 102to analyze the information on the posture. In addition, thecontrol-information generation section 72 also generates controlinformation, which will be used for controlling the balance between thevolumes of the voices of the users X and Y in the synthesis of sounds,on the basis of a result of a process carried out by theposture-information analysis section 102 to analyze the information onthe posture.

The reference head position H0 is set in advance by theposture-information analysis section 102. Typically, the reference headposition H0 is set in the same way as the method of setting a referencecontour G0 as explained earlier by referring to FIG. 8 as follows. Firstof all, images of the user A having a reference posture are taken byusing the camera 51. The reference posture is a reference posture inwhich the reference head position H0 is set as shown as a middle postureat the bottom of FIG. 9. From images taken by the camera 51, thereference posture of the user A is then recognized by adoption of amethod such as a motion vector analysis. Finally, the information on thereference head position in the recognized reference posture of the userA is acquired.

The posture-information analysis section 102 analyzes the head positionH1 of the user A in the current posture by comparing the head positionH1 with the reference head position H0 of the user A. In the diagram forshowing a graph 232 in FIG. 10, the reference head position H0corresponds to a value of 0 on the horizontal axis.

The vertical axis of the right-side diagram showing the graph 232 inFIG. 10 represents the information on the posture. The center of thevertical axis is the posture information corresponding to the referenceposture. Posture information above the center represents right-leaningpostures and posture information below the center representsleft-leaning postures. On the other hand, the horizontal axis representsthe current head position H1 relative to the reference head position H0.The center of the horizontal axis is the reference head position H0 of0. Head positions H1 on the right side of the center are head positionsH1 greater than the reference head position H0, each representing aposition on the + side corresponding to right-leaning postures. On theother hand, head positions H1 on the left side of the center are headpositions H1 smaller than the reference head position H0, eachrepresenting a position on the − side corresponding to left-leaningpostures. It is to be noted that, in an image 231 shown in FIG. 10 as animage of the user A, the left and right directions are opposite to thedirections of the user A viewing and listening to the display 41 shownin FIG. 9. That is to say, the − side of the image 231 is the − side ofthe horizontal axis of the graph 232 shown in FIG. 10, representing theleft side of FIG. 9 and the + side of the image 231 is the + side of thehorizontal axis of the graph 232, representing the right side of FIG. 9

Thus, if the current head position H1 is smaller than 0, that is, if thecurrent head position H1 of the user A is a position on the − side ofthe reference position H0, the posture-information analysis section 102determines that the posture of the user A is left leaning with respectto the reference posture, generating an analysis result indicating thatthe user A is facing the screen in a direction toward the user Ydisplayed on the aforementioned child screen 222 on the left side on thedisplay unit 41 showing the image of the content. In this case, on thebasis of the analysis result, the control-information generation section72 generates control information to be used for synthesizing images todisplay a child screen 222L at a size larger than the size of the childscreen 222 as shown in the child screen 222L and control information tobe used for synthesizing sounds to output the voice of the user Y at alarger volume. In addition, at the same time, the control-informationgeneration section 72 generates control information to be used forsynthesizing images to display a child screen 221S at a size smallerthan the size of the child screen 221 cited earlier as shown in thechild screen 221S and control information to be used for synthesizingsounds to output the voice of the user X at a smaller volume. In thisway, the user A can view and listen to the user Y better than the user Aviews and listens to the user X.

If the current head position H1 is greater than 0, that is, if thecurrent head position H1 of the user A is a position on the + side ofthe reference position H0, the posture-information analysis section 102determines that the posture of the user A is right leaning with respectto the reference posture, generating an analysis result indicating thatthe user A is facing the screen in a direction toward the user Xdisplayed on the aforementioned child screen 221 on the right side onthe display unit 41 showing the image of the content. In this case, onthe basis of the analysis result, the control-information generationsection 72 generates control information to be used for synthesizingimages to display a child screen 221L at a size larger than the size ofchild screen 221 as shown in the child screen 221L and controlinformation to be used for synthesizing sounds to output the voice ofthe user X at a larger volume. In addition, at the same time, thecontrol-information generation section 72 generates control informationto be used for synthesizing images to display a child screen 222S at asize smaller than the size of the child screen 222 mentioned before asshown in the child screen 222S and control information to be used forsynthesizing sounds to output the voice of the user Y at a smallervolume. In this way, the user A can view and listen to the user X betterthan the user A views and listens to the user Y.

As described above, in the case of the examples shown in FIGS. 9 and 10,if the user A has a left-leaning posture, the synthesis is carried outto produce a display with the image of the user Y striking and the imageof the user X not striking. Thus, the user A is capable of easilyobtaining an environment allowing a communication with the user Y to becarried out simply since the expression, body signal language andgesture of the user Y having a conversation with the user A can be seenwith ease. If the user A has a right-leaning posture, on the other hand,the synthesis is carried out to produce a display with the image of theuser X striking and the image of the user Y not striking. Thus, the userA is capable of easily obtaining an environment allowing a communicationwith the user X to be carried out simply since the expression, bodysignal language and gesture of the user X having a conversation with theuser A can be seen with ease.

It is thus possible to provide the user with the same feeling as anactual interactive communication in which the user is facing and talkingto a communication partner having a conversation with the user. As aresult, a natural and lively communication can be provided.

In the case of the example shown in FIG. 10, the information on theleft-leaning and right-leaning postures of the user A is analyzed byextracting the head position from an image of the user A. It is to benoted, however, that the information on the left-leaning andright-leaning postures of the user A can also analyzed by, for example,detecting information on the line of vision and information on thelayout of the eyes, nose and mouth of the user A.

By referring to the examples shown in FIGS. 7 to 10, the abovedescriptions have explained analysis processes using user-conditioninformation included in real-time data of the user A. In the case of theexamples, the information on the condition of a user is an image of theuser. However, the communication apparatus 1-1 operated by the user Acan also be controlled by analyzing user condition information obtainedfrom real-time data of another user serving as a communication partneroperating another communication apparatus installed at a remote locationin addition to user condition information obtained from real-time dataof the user A. In this case, the real-time data of the other user isreceived from the other communication apparatus operated by the otheruser.

By referring to FIGS. 11 and 12, the following description concretelyexplains a method to analyze information on conditions of users by usinga plurality of pieces of real-time data. It is to be noted that examplesshown in FIGS. 11 and 12 are typical processing carried out by thecontrol-information generation section 72 to generate controlinformation on the basis of results of analyses carried out by theuser-count analysis section 103 and the audio-information analysissection 104.

In the example shown in FIG. 11, the user A utilizes the communicationapparatus 1-1 to adopt the picture-in-picture method explained earlierby referring to FIG. 3A as a method for carrying out the remotecommunication recording process at the step S5 of the flowchart shown inFIG. 5 to communicate with the user X operating the communicationapparatus 1-2, the user Y operating another communication apparatus 1-3shown in none of the figures and a user Z operating a furthercommunication apparatus 1-4 also shown in none of the figures. Thus, 4users are involved in the remote communication recording process.

In this case, the input section 22 employed in the communicationapparatus 1-1 is used to acquire real-time data including an image andvoice of the user A, brightness and an ambient temperature and transmitthe data to the communication apparatus 1-2 to 1-4 by way of thecommunication section 23. On the other hand, the communication apparatus1-2 to 1-4 transmit the real-time data of the users X, Y and Z to thecommunication section 23 employed in the communication apparatus 1-1. Itis to be noted that FIG. 11 shows a microphone 52 as the input section22.

Then, the display unit 41 employed in the communication apparatus 1-1displays an image of the user Y serving as a communication partner on achild screen 252 superposed on the image 254 of a content beingreproduced on the right lower corner on the image 254, an image of theuser X serving as another communication partner on a child screen 251superposed on the image 254 on the middle lower corner of the image 254and an image of the user Z serving as a further communication partner ona child screen 253 superposed on the image 254 on the left lower cornerof the image 254.

At that time, the user A carries out an operation to start theuser-characteristic analysis mixing process by operating the operationinput section 31.

The user-characteristic analysis section 71 has received real-time dataof the users A, X, Y and Z. The user-count analysis section 103 employedin the user-characteristic analysis section 71 analyzes the data todetermine how many users are participating in the remote communicationprocessing. In addition, the audio-information analysis section 104employed in the user-characteristic analysis section 71 extracts audioinformation (or audio characteristics) such as volumes and frequencycharacteristics from sounds included in the real-time data of the usersA, X, Y and Z. Then, the audio-information analysis section 104 analyzesthe extracted volumes and frequency characteristics to determine thelevel of liveliness for each of the users participating in thecommunication, that is, the liveliness indicator of each user.Subsequently, on the basis of the determination result, thecontrol-information generation section 72 generates control informationto be used for controlling the size, degree of transparency and screeneffect for each of the child screens 251 to 253 synthesized in the imageof the content.

The horizontal axis of a diagram showing graphs 261 in FIG. 12represents the lapse of time. The vertical axis represents the volume ofa voice output by each of the users A, X, Y and Z. The graphs 261arranged from the top to the bottom represent changes of the volumes ofvoices output by the users A, X, Y and Z respectively with the lapse oftime. The higher the position of a graph, the larger the volumerepresented by the graph. By the same token, the horizontal axis of adiagram showing graphs 262 in FIG. 12 also represents the lapse of time.The vertical axis represents the relative level of liveliness for eachof the users A, X, Y and Z at the predetermined time. The relative levelof liveliness for a user is inferred on the basis of the volume of avoice output by the user. The longer the distance between a relativelevel of liveliness above the center of the vertical axis and thecenter, the higher the relative level of liveliness.

That is to say, as shown by the graphs 261, the volume of a voice outputby the user A prior to a time t0 is smaller than the volumes of voicesoutput by the users Y and Z. As shown by portions ‘a’ after the time t0,however, there are many periods of time in which the volume of a voiceoutput by the user A exceeds the volumes of voices output by the users Yand Z. Thus, an analysis result indicates that, prior to the time t0 inthe graphs 262, the level of liveliness for the user A is lower than areference level but, after the time to, the level of liveliness for theuser A is higher than the reference level.

Also as shown by the graphs 261, a state in which the volume of a voiceoutput by the user X is smaller than the volumes of voices output by theusers A, Y and Z is continuous prior to the time t0. As shown by aportion ‘x’ after the time t0, however, a state in which the volume of avoice output by the user X is larger than the volumes of voices outputby the users A, Y and Z is continuous for a long period of time. Thus,an analysis result indicates that, prior to the time t0 in the graphs262, the level of liveliness for the user X is lower than not only thereference level, but also the levels of liveliness for the users A, Yand Z but, after the time to, the level of liveliness for the user X ishigher than not only the reference level, but also the levels ofliveliness for the users A, Y and Z.

Also as shown by the graphs 261, prior to the time t0, there are manyperiods of times in which the volume of a voice output by the user Y islarger than the volumes of voices output by the users A and X. Exceptfor 2 portions ‘y’ each showing a volume of a voice output by the user Ylarger than the volumes of voices output by the users A and X, however,a state in which the volume of a voice output by the user Y is smallerthan the volumes of voices output by the users A and X is continuousafter the time t0. Thus, an analysis result indicates that, prior to thetime t0 in the graphs 262, the level of liveliness for the user Y ishigher than not only the reference level but also the levels ofliveliness for the users A and X but, after the time to, the level ofliveliness for the user A is lower a little bit than the referencelevel.

Also as shown by the graphs 261, a state in which the volume of a voiceoutput by the user Z is larger than the volumes of voices output by theusers A, X and Y is continuous prior to the time t0. As shown by aportion ‘z’ after the time to, however, a state in which the volume of avoice output by the user Z is smaller than the volumes of voices outputby the users A, X and Y is continuous for a long period of time. Thus,an analysis result indicates that, prior to the time t0 in the graphs262, the level of liveliness for the user Z is higher than not only thereference level, but also the levels of liveliness for the users A, Xand Y but, after the time t0, the level of liveliness for the user Z islower than not only the reference level, but also the levels ofliveliness for the users A, X and Y.

If an analysis result indicates that a state in which the user X iscontinuously outputting a voice at a largest volume in comparison withthose of the users A, Y and Z is sustained, the control-informationgeneration section 72 generates control information to be used forsynthesizing images in such a way that the size of the child screen 251showing the image of the user X is made larger than the sizes of theother child screens 252 and 253 as shown by an arrow P1 in FIG. 11. Ifan analysis result indicates that a state in which the user Z iscontinuously outputting a voice at a small volume in comparison withthose of the users A, X, and Y or an almost quiet state of the user Z issustained, the control-information generation section 72 generatescontrol information to be used for synthesizing images in such a waythat the size of the child screen 253 showing the image of the user Z ismade smaller or thinner than the sizes of the other child screens 251and 252 as shown by an arrow P2.

Let us keep in mind that, for example, it is also possible to provide aconfiguration in which frequency characteristics are extracted fromsounds included in real-time data of the users A, X, Y and Z and, if ananalysis result of the extracted frequency characteristics indicatesthat a state in which the frequency characteristic of the user X iscontinuously high in comparison with those of the users A, Y and Z issustained, the control-information generation section 72 generatescontrol information to produce a screen effect of typically blinking thechild screen 251 showing the image of the user X.

As described above, in the examples shown in FIGS. 11 and 12, theuser-characteristic analysis section 71 analyzes not only audioinformation extracted from a voice of the user A itself, but also audioinformation extracted from voices of communication partners. Inaddition, on the basis of the analysis result, a synthesis of childscreens each showing a user is controlled to typically increase the sizeof the child screen showing a user at the highest level of livelinessamong the users participating in a communication, that is, the childscreen showing a most excited user. Thus, it is possible to provide anenvironment allowing a communication according to the transmissioncondition existing at that time to be obtained with ease. It is ofcourse possible to provide a configuration in which theuser-characteristic analysis section 71 does not analyze audioinformation extracted from a voice output by the user A itself, butanalyzes only audio information extracted from voices of communicationpartners.

In addition, in the examples shown in FIGS. 11 and 12, it is possible toprovide a configuration in which, besides the information used tocontrol the synthesis of displayed child screens, control informationshowing results of comparisons of voices output by the users is alsogenerated. While the examples shown in FIGS. 11 and 12 are referred toin explanations of syntheses carried out on the basis of voices, theexamples can also be referred to in explanations of syntheses carriedout on the basis of other user condition information such as images. Inthe case of syntheses carried out on the basis of images, for example,information on the line of vision for each other user is extracted froman image included in real-time data of the other user and, on the basisof the extracted information on the line of vision for each of otherusers, it is possible to analyze the child screens of the users in aconversation and other attributes of the users.

It is to be noted that, as is obvious from the examples shown in FIGS.11 and 12, an analysis of user condition information obtained fromreal-time data of communication partners can be said in other words tobe control executed in accordance with behaviors of the communicationpartners and communication environments as control of the communicationapparatus 1-1 operated by the user A. In consequence, an operation andeffect not desired by the user A may be resulted in inadvertently. Inorder to get rid of such an operation and effect, as will be describedlater by referring to FIG. 13, the communication apparatus 1-1 operatedby the user A can be set not to analyze real-time data of the otherusers or to limit control information generated from results of otheranalyses carried out on the real-time data of the other users.

In the above descriptions, the picture-in-picture method is assumed.However, the scope of the present invention is not limited to thepicture-in-picture method. That is to say, the present invention canalso be applied to the cross fade method explained earlier by referringto FIG. 3B, the wipe method explained before by referring to FIG. 3C andother synthesis patterns.

In addition, the above descriptions explain only syntheses of an imageand voice of each communication partner with an image and sound of acontent. However, an image and voice input by the input section 22 as animage and voice of the user A can also be synthesized with an image andsound of a content.

Next, the user-characteristic analysis mixing process carried out at thestep S6 of the flowchart shown in FIG. 5 is explained by referring to aflowchart shown in FIG. 13 as follows.

At the step S5 of the flowchart shown in FIG. 5, acommunication-recording process is started. Then, on the basis of asynthesis pattern set in advance by an operation carried out by the userand synthesis parameters also set in advance by an operation carried outby the user for the synthesis pattern, the synthesis control section 84carries out a process to control the synthesis processing performed bythe audio/video synthesis section 26. In addition, the data analysissection 28 obtains a reproduced content, input real-time data of theuser A and other users and received real-time data of the user X.

Then, the user A operates the operation input section 31 to enter acommand making a request for a start of the user-characteristic analysismixing process. The operation input section 31 generates an operationsignal corresponding to the operation carried out by the user A andsupplies the operation signal to the synthesis control section 84.Receiving the operation signal from the operation input section 31, atthe first step S21 of the flowchart shown in FIG. 13, the synthesiscontrol section 84 produces a determination result as to whether or notto start the user-characteristic analysis mixing process. If thedetermination result indicates that the user-characteristic analysismixing process is to be started, the flow of the processing goes on to astep S22 at which the synthesis control section 84 produces adetermination result as to whether or not real-time data of other usersis to be analyzed. If the determination result indicates that real-timedata of other users is not to be analyzed, the flow of the processinggoes on to a step S23.

Information as to whether or not real-time data of other users is to beanalyzed has been set prior to the user-characteristic analysis mixingprocess. It is needless to say that, by operating the operation inputsection 31, the user A is capable of changing the setting of theinformation indicating whether or not real-time data of other users isto be analyzed.

Then, at the step S23, in accordance with control executed by thesynthesis control section 84, the analysis control section 101 controlsone of the user-count analysis section 103, the audio-informationanalysis section 104 and the environment-information analysis section105 to analyze user condition information (or user characteristics)obtained from real-time data of the user itself, who is the user A, andsupply the analysis result to the control-information generation section72. Then, the flow of the processing goes on to a step S24.

At the step S24, the control-information generation section 72 generatescontrol information on the basis of the analysis result received fromthe analysis control section 101, supplying the control information tothe synthesis control section 84, the operation-information outputsection 87 or the electronic-apparatus control section 88. Then, theflow of the processing goes on to a step S28 at which images and voicesare synthesized in accordance with the control information.

The processes carried out at the steps S23 and S24 are explained inconcrete terms as follows. For example, at the step S23, the analysiscontrol section 101 controls the posture-information analysis section102 to extract information on a posture from an image included in thereal-time data of the user A, to analyze the information on a posture inorder to determine whether the posture of the user is a forward,backward, left or right-leaning posture as explained earlier byreferring to FIGS. 7 to 10 and to supply the analysis result to thecontrol-information generation section 72. On the basis of the analysisresult, at the next step S24, the control-information generation section72 generates control information to be used for controlling theaudio/video synthesis section 26 for synthesizing the image and sound ofa content with the image and voice of the user X serving as acommunication partner and supplies the control information to thesynthesis control section 84. In this case, the control information isinformation used for setting a synthesis pattern and synthesisparameters for the pattern.

As another example, at the step S23, the analysis control section 101controls the environment-information analysis section 105 to analyzebrightness information included the real-time data of the user A inorder to produce a determination result as to whether the environment ofthe user A is dark or bright and supply the determination result to thecontrol-information generation section 72. If the determination resultproduced by the environment-information analysis section 105 inaccordance with the control executed by the analysis control section 101indicates that the environment of the user A is dark, at the next stepS24, the control-information generation section 72 generates controlinformation to make the setting of an illumination apparatus brighterand supplies the control information to the electronic-apparatus controlsection 88. Normally, the illumination apparatus is provided as aperipheral of the communication apparatus 1-1 as shown in none of thefigures.

In accordance with the control information, at a step S29, theelectronic-apparatus control section 88 puts the illumination apparatusat bright setting in accordance with the control information receivedfrom the control-information generation section 72. In this way, forexample, it is possible to get rid of a situation in which the user isviewing a content such as an animation appearing on the display unit 41in a dark environment because the user is too much focusing attention onthe content so that the user has forgotten to make the room bright and,as a result, the user feels unwell.

If the determination result produced at the step S22 indicates thatreal-time data of other users is also to be analyzed, on the other hand,the flow of the processing goes on to a step S25.

At the step S25, in accordance with control executed by the synthesiscontrol section 84, the analysis control section 101 controls one of theuser-count analysis section 103, the audio-information analysis section104 and the environment-information analysis section 105 to analyze usercondition information (or user characteristics) obtained from real-timedata of the user itself, who is the user A, and condition informationobtained from real-time data of the other users each serving as acommunication partner, supplying the analysis result to thecontrol-information generation section 72. Then, the flow of theprocessing goes on to a step S26.

At the step S26, the control-information generation section 72 generatescontrol information on the basis of the analysis result received fromthe analysis control section 101. Then, at the next step S27, thecontrol-information generation section 72 produces a determinationresult as to whether or not the generated control information is to berejected. If the determination result indicates that the generatedcontrol information is not to be rejected, the control-informationgeneration section 72 supplies the generated control information to thesynthesis control section 84, the operation-information output section87 or the electronic-apparatus control section 88. Then, the flow of theprocessing goes on to the step S28 cited before. That is to say, ifcontrol to be executed on the basis of an analysis result carried out onreal-time data of the communication partners will cause an operationand/or effect not desired by the user, the generated control informationcan be rejected.

The processes carried out at the steps S25 to S27 are explained inconcrete terms as follows. For example, at the step S25, the analysiscontrol section 101 controls the user-count analysis section 103 and theaudio-information analysis section 104 to extract the number ofcommunication partners and audio information such as volumes andfrequency characteristics from sounds included in the real-time data ofthe user A and the real-time data of the communication partners, analyzethe extracted audio information in order to determine the level ofliveliness for each of the users participating in the communication,that is, the liveliness indicator of each user as explained earlier byreferring to FIGS. 11 and 12 and supply the analysis result to thecontrol-information generation section 72. Then, at the next step S26,on the basis of the analysis result, the control-information generationsection 72 generates control information to be used for controlling theaudio/video synthesis section 26 for synthesizing the image and sound ofa content with the image and voice of each of the users X, Y and Z eachserving as a communication partner. In this case, the controlinformation is information used for setting a synthesis pattern andsynthesis parameters for the pattern. If the generated controlinformation is found at the next step S27 to be not information to berejected, the control-information generation section 72 supplies theinformation to the synthesis control section 84.

As another example, at the step S25, the analysis control section 101controls the audio-information analysis section 104 to analyticallycompare an environment sound extracted from the real-time data of theuser A as a sound of an environment surrounding the user A with a voiceextracted from the real-time data of a communication partner as a spokenvoice of the communication partner, and supply the result of thecomparison to the control-information generation section 72. If thecomparison result produced by the audio-information analysis section 104in accordance with the control executed by the analysis control section101 indicates that the spoken voice of the communication partner issofter than the environment sound of the user A, at the next step S26,the control-information generation section 72 generates controlinformation to increase the volume of a sound output by the speaker 42and supplies the control information to the electronic-apparatus controlsection 88 if the generated control information is found at the nextstep S27 to be not information to be rejected. In this way, the spokenvoice of the communication partner can be heard better.

Let us keep in mind that it is also possible to provide a configurationin which, at that time, the environment sound of a communication partnersuch as the user X is further compared analytically with the spokenvoice of the user A at the step S25. If the spoken voice of the user Ais found softer than the environment sound of the user X, at the nextstep S26, the control-information generation section 72 generatescontrol information to increase the volume of a sound output by thespeaker 42 employed in the communication apparatus 1-2 and supplies thecontrol information to the operation-information output section 87 to betransmitted to the communication apparatus 1-2 at a step S30 to bedescribed later. In this way, the spoken voice of the user A can beheard better by the communication partner. In addition, in place of thecontrol information to increase the volume of a sound output by thespeaker 42, control information to increase the volume of a synthesizedsound can be generated instead.

Receiving the control information, at the step S29 cited before, theelectronic-apparatus control section 88 puts the speaker 42 at settingto increase the volume of a sound output by the speaker 42 in accordancewith the control information received from the control-informationgeneration section 72. In this way, when a user in a noisy environmentnaturally speaks loudly due to the noisy environment, generating audioinformation indicating a loud voice of the user, for example, acommunication apparatus operated by a communication partner in a quieterenvironment is capable of reducing the volume of the voice of the userin accordance with the audio information extracted by the communicationapparatus. Conversely, when a user in a quiet environment naturallyspeaks softly due to the quiet environment, generating audio informationindicating a soft voice of the user, on the other hand, a communicationapparatus operated by a communication partner in a noisier environmentis capable of increasing the volume of the voice of the user inaccordance with the audio information extracted by the communicationapparatus. As a result, it is possible to implement a communicationsuitable for the environments of the users with ease.

As a further example, at the step S25, the analysis control section 101controls the posture-information analysis section 102 to extractinformation on a posture of a communication partner from images includedin the real-time data of the user A and the communication partner,analyze the information on a posture and supply the analysis result tothe control-information generation section 72. If the analysis resultproduced by the posture-information analysis section 102 in accordancewith the control of the analysis control section 101 indicates that thecontour of the communication partner exceeds a reference contour, thatis, if the analysis result indicates that the communication partner hascome closer to the display unit 41 employed in the communicationapparatus 1-2, at the next step S26, the control-information generationsection 72 generates control information to be used for synthesizingimages in such a way that the size of a child screen for displaying thecommunication partner is increased, and supplies the control informationto the synthesis control section 84 if the generated control informationis found at the next step S27 to be not information to be rejected.

Receiving the control information, the step S28 to be described in moredetail bellow, the synthesis control section 84 controls the licensestorage section 62 in accordance with the control information receivedfrom the control-information generation section 72. Thus, it is possibleto provide an effect of making the user A interested in the fact thatthe communication partner has come closer to the display unit 41employed in the communication apparatus 1-2.

The step S28 of the flowchart shown in FIG. 13 is described in moredetail as follows. At this step, in accordance with the controlinformation received from the control-information generation section 72,the synthesis control section 84 sets a synthesis pattern of theaudio/video synthesis section 26 and synthesis parameters for thesynthesis pattern. Then, the audio/video synthesis section 26synthesizes the image and sound of the content with respectively theimage and voice of a communication partner. Then, the flow of theprocessing goes on to the step S29 described earlier.

Thus, the display unit 41 employed in the output section 21 shows animage of the content and an image of a user serving as a communicationpartner as a result of an image synthesis according to controlinformation generated by the control-information generation section 72on the basis of an analysis result carried out by theuser-characteristic analysis section 71. On the other hand, the speaker42 employed in the output section 21 outputs a sound of the content anda voice of a user serving as a communication partner as a result of asound synthesis according to control information generated by thecontrol-information generation section 72 on the basis of an analysisresult carried out by the user-characteristic analysis section 71.

In addition, a synthesis pattern updated in accordance with controlinformation generated by the control-information generation section 72and synthesis parameters also updated in accordance with the controlinformation as parameters for the synthesis pattern are recorded assynthesis information along with the content, the reproduction of whichhas been started, the images included in the input real-time data of theuser A and the other input real-time data as well as the image includedin the received real-time data of the user X, the voices included in theinput real-time data of the user A and the other input real-time data aswell as the voice included in the received real-time data of the user X.

At the step S29 described before, in accordance with the controlinformation received from the control-information generation section 72,the electronic-apparatus control section 88 executes control of sectionscomposing the communication apparatus 1-1 and electronic apparatus eachserving as a peripheral of the communication apparatus 1-1. For example,the electronic-apparatus control section 88 adjusts the volume set forthe speaker 42 and makes the setting of a peripheral illuminationapparatus brighter. Then, the flow of the processing goes on to the stepS30 mentioned before.

At the step S30, the operation-information output section 87 transmitscontrol information received from the control-information generationsection 72 as control information for the communication apparatus 1-2operated by the user X to the communication apparatus 1-2 by way of thecommunication section 23 and the communication network 2. Then, the flowof the processing goes on to a step S31. It is to be noted thatprocessing carried out by the communication apparatus 1-2 receiving thecontrol information from the communication apparatus 1-1 will bedescribed later.

The user A may operate the operation input section 31 to enter a commandmaking a request for an end of the user-characteristic analysis mixingprocess. The operation input section 31 generates an operation signalcorresponding to the operation carried out by the user A and suppliesthe operation signal to the synthesis control section 84. At the nextstep S31, on the basis of such an operation signal, the synthesiscontrol section 84 produces a determination result as to whether or notto end the user-characteristic analysis mixing process. If thedetermination result indicates that the user-characteristic analysismixing process is to be ended, the user-characteristic analysis mixingprocess is terminated and the flow of the processing goes back to thestep S6 included in the flowchart shown in FIG. 5 as a step followingthe step S7.

By the same token, if the determination result produced at the step S21indicates that the user-characteristic analysis mixing process is not tobe started or if the determination result produced at the step S27indicates that the generated control information is information to berejected, the user-characteristic analysis mixing process is terminatedand the flow of the processing goes back to the step S6 included in theflowchart shown in FIG. 5 as a step following the step S7 as is the casewith the determination result of the step S31 indicating that theuser-characteristic analysis mixing process is to be ended. That is tosay, at the step S7, the synthesis control section 84 continues toperform processing of controlling a synthesis process carried out by theaudio/video synthesis section 26 on the basis of a synthesis pattern setin advance in accordance with an operation performed by the user andsynthesis parameters set in advance also in accordance with an operationdone by the user as parameters for the pattern till the user executes anoperation to make a request for termination of the remote communication.If the determination result produced at the step S31 indicates that theuser-characteristic analysis mixing process is not to be ended, on theother hand, the flow of the processing goes back to the step S22.

As described above, user condition information including an image,voice, brightness and temperature extracted from real-time data of usersis analyzed and, on the basis of the analysis result, a synthesis of animage and sound of a content with an image and voice of a communicationpartner or setting of sections composing a control apparatus iscontrolled. It is thus possible to implement a communication reflectingconditions of the users in a real-time manner. As a result, it ispossible to provide an effect of a face-to-face communication in spiteof the fact that the users are present at locations remote from eachother.

In addition, since it is also possible to easily implement various kindsof setting of these complicated communication apparatus, which aredifficult for the user to set, as setting desired by the user, the laborfor the setting can be eliminated.

A communication apparatus operated by a communication partner can alsobe controlled.

By referring to a flowchart shown in FIG. 14, the following descriptionexplains a control-information receiver process carried out by thecommunication apparatus 1-2 to receive control information transmittedby the communication apparatus 1-1 at the step S30 of the flowchartshown in FIG. 13.

It is to be noted that the control-information receiver processrepresented by the flowchart shown in FIG. 14 is a process, which iscarried out by the communication apparatus 1-2 while theremote-communication recording process is being performed after the stepS5 of the flowchart shown in FIG. 5. That is to say, thecontrol-information receiver process is processing to implement a mixingprocess in accordance with another result provided by the communicationapparatus 1-1 as a result of another analysis of contentcharacteristics. In other words, the control-information receiverprocess is another type of content-characteristic analysis mixingprocess carried out at the step S6 of the flowchart shown in FIG. 5.

The flowchart shown in FIG. 14 begins with a step S51 at which thecommunication section 23 employed in the communication apparatus 1-2receives control information from the operation-information outputsection 87 employed in the communication apparatus 1-1 and supplies thecontrol information to the session management section 81.

Then, at the next step S52, the session management section 81 outputs adetermination result as to whether or not the control informationreceived from the communication apparatus 1-1 is information to berejected by determining whether or not the control information isinformation causing an operation and/or effect not desired by the user.If the determination result indicates that the control informationreceived from the communication apparatus 1-1 is information to berejected, the control-information receiver processing is ended.

Let us keep in mind that it is also possible to set the communicationapparatus 1-2 to optionally accept or reject control informationreceived from the communication apparatus 1-1 or always reject suchinformation. In addition, it is also possible to provide a configurationin which, if control information is accepted in the communicationapparatus 1-2, the communication apparatus 1-2 itself analyzes theinformation and priority levels are set for exclusive execution ofgenerated control information or a master-slave relation is set inadvance among the communication apparatus.

If the determination result produced by the session management section81 at the step S52 indicates that the control information received fromthe communication apparatus 1-1 is not information to be rejected, onthe other hand, the control information is supplied to the synthesiscontrol section 84 or the electronic-apparatus control section 88. Then,the flow of the processing goes on to a step S53.

At the step S53, the synthesis control section 84 sets a synthesispattern for the audio/video synthesis section 26 and synthesisparameters for the synthesis pattern in accordance with controlinformation received from the control-information generation section 72.Then, the synthesis control section 84 controls the audio/videosynthesis section 26 to synthesize an image and sound of the contentwith the image and voice of each user serving as a communicationpartner. Subsequently, the flow of the processing goes on to a step S54.

At the step S54, the electronic-apparatus control section 88 controlssections employed in the communication apparatus 1-2 and electronicapparatus each serving as a peripheral of the communication apparatus1-2 or other equipment in accordance with the control information.Finally, the execution of the processing represented by this flowchartis ended.

As described above, it is possible to use not only control informationgenerated by the control-information generation section 72 in accordancewith an analysis result carried out by the user-characteristic analysissection 71 employed in the communication apparatus itself, but alsocontrol information generated by the control-information generationsection 72 in accordance with an analysis result carried out by theuser-characteristic analysis section 71 employed in anothercommunication apparatus. In addition, the control information can alsobe rejected.

It is to be noted that the above descriptions assume that eachcommunication apparatus includes a data analysis section 28. However, aserver including the data analysis section 28 may also be connected tothe communication network 2 to serve as an apparatus for providingcontrol information to each communication apparatus. As an alternative,the server can also be provided with only the user-characteristicanalysis section 71 so that the server is capable of giving analysisinformation to each communication apparatus.

Since remote communication processing is carried out as described above,more lively and natural communications can be implemented in comparisonwith equipment including the telephone set in related art, the TVtelephone set and remote communication apparatus such as the videoconference system.

That is to say, in the case of the communication in related art, theuser X using a TV set in related art to view and listen to a broadcastcontent distributed in a real-time manner utilizes an audio telephoneset to express an impression of the broadcast content viewed andlistened to by the user X to the user A present at a remote location. Inthis case, nevertheless, it is difficult for the user A, who does notactually view and listen to the broadcast content, to understand theimpression of the situation.

By using the communication apparatus according to an embodiment of thepresent invention, however, the users A and X present at locationsremote from each other are capable of sharing the content at the sametime and, in addition, the images of the users A and X can be reproducedon child screens or the like while their voices can be heard. Thus, inspite of the fact that the users A and X are present at locations remotefrom each other, it is possible to provide a high realistic sensation, asense of togetherness and a sense of intimacy as if a face-to-facecommunication were being carried out.

By using video information extracted from an image of a user, usingaudio information extracted from a sound output by the user, usingenvironment information extracted from the brightness and temperature ofan environment surrounding the user and using other kinds ofinformation, processing such as a process to synthesize the image andsound of a content and the image and sound of the user can becontrolled. Thus, parameters of a communication apparatus can be seteasily in accordance with conditions such as a situation faced by theuser and the state of communication without taking much time and labor.As a result, more lively and natural communications can be implemented.In this case, the video information cited above includes a posture andface orientation of the user whereas the audio information mentionedabove includes resounding cheers of the user and its claps.

The series of processes carried out by the communication apparatus 1 asdescribed previously can be carried out by hardware and/or execution ofsoftware. In this case, each of the communication apparatus 1-1 and 1-2shown in FIG. 1 is typically implemented by a personal computer 401 likeone shown in FIG. 15.

In the personal computer 401 shown in FIG. 15, a CPU (Central ProcessingUnit) 411 is a component for carrying out various kinds of processing byexecution of a variety of programs stored in advance in a ROM (Read OnlyMemory) 412 or loaded into a RAM (Random Access Memory) 413 from astorage section 418. The RAM 413 is also used for properly storing databy the CPU 411 in the executions of the programs.

The CPU 411, the ROM 412 and the RAM 413 are connected to each otherthrough a bus 414. The bus 414 is also connected to an input/outputinterface 415.

The input/output interface 415 is connected to an input section 416, anoutput section 417, the storage section 418 mentioned above and acommunication section 419. Used for receiving a command entered by theuser, the input section 416 includes input devices such as a keyboardand a mouse, whereas the output section 417 includes a display unit fordisplaying an image and a speaker for outputting a generated sound. Thedisplay unit is typically a CRT (Cathode Ray Tube) display unit or anLCD (Liquid Crystal Display) unit. The storage section 418 is typicallya hard-disk drive including an embedded hard disk used for storing avariety of programs and various kinds of data. The communication section419 including a modem and a terminal adapter is a unit for carrying outradio or wire communication processing with other apparatus through anetwork.

The input/output interface 415 is also connected to a drive 420 on whicha recording medium is mounted. Examples of the recording medium are amagnetic disk 421, an optical disk 422, a magneto-optical disk 423 and asemiconductor memory 424. If necessary, a program read out from therecording medium is installed in the storage section 418.

As explained above, the series of processes carried out by thecommunication apparatus 1 as described previously can be carried out byhardware and/or execution of software. If the series of processesdescribed above is carried out by execution of software, programscomposing the software can be installed into a computer embedded indedicated hardware, a general-purpose personal computer or the like fromtypically a network or the recording medium described above. In thiscase, the computer or the personal computer serves as the communicationapparatus 1 described above. By installing a variety of programs intothe general-purpose personal computer, the personal computer is capableof carrying out a variety of functions. A typical general-purposepersonal computer 401 is shown in FIG. 15 as described above.

As explained above, if necessary, a program read out from the recordingmedium as the software mentioned above is installed in the storagesection 418. The recording medium itself is distributed to usersseparately from the main unit of the communication apparatus 1. As shownin FIG. 15, examples of the recording medium also referred to as packagemedia are magnetic disks 421 including a flexible disk, optical disks422 including a CD-ROM (Compact Disk-Read Only Memory) and a DVD(Digital Versatile Disk), magneto-optical disks 423 including an MD(Mini Disk) and a semiconductor memory 424. As an alternative toinstallation of a program from the package media into the storagesection 418, the program can also be stored in advance typically in theROM 412 or a hard disk embedded in the storage section 418.

It is worth noting that, in this specification, steps of any programrepresented by a flowchart described above can be carried out not onlyin a pre-prescribed order along the time axis, but also concurrently orindividually.

It is also to be noted that the technical term ‘system’ used in thisspecification implies the configuration of a confluence including aplurality of apparatus.

It should be understood by those skilled in the art that a variety ofmodifications, combinations, sub-combinations, and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

1. An information-processing apparatus for communicating with an otherinformation-processing apparatus, which is connected to saidinformation-processing apparatus by a network, saidinformation-processing apparatus comprising: a reproducer configured toreproduce moving video content data common to saidinformation-processing apparatus and said other information-processingapparatus synchronously with said other information-processingapparatus; an input unit configured to input first user conditioninformation including a voice and image of a user operating saidinformation-processing apparatus; a receiver configured to receivesecond user condition information including a voice and image of another user from said other information-processing apparatus operated bysaid other user; a synthesizer configured to synthesize a voice andimage of said moving video content data synchronously reproduced by saidreproducer with said voice and image of said other user received by saidreceiver by superimposing the voice and image onto the moving videocontent data, the synthesizer creating a single moving image includingboth the voice and image of the moving video content data and the voiceand image of the other user such that the image of the moving videocontent data extends within an area and at least part of the image ofthe other user is included within the area of the image of the movingvideo content data; a display configured to simultaneously display saidvoice and image of said content data synthesized with said voice andimage of said other user; an analyzer configured to analyze at leasteither said first user condition information input by said input unit orsaid second user condition information received by said receiver; and aparameter setting unit configured to set a control parameter used forcontrolling a process, which is carried out by said synthesizer tosynthesize voices and images on the basis of an analysis result producedby said analyzer.
 2. The information-processing apparatus according toclaim 1 wherein said first user condition information also includesinformation on the environment of said user whereas said second usercondition information also includes information on the environment ofsaid other user.
 3. The information-processing apparatus according toclaim 1, further comprising: a generator configured to generate adetermination result as to whether or not said second user conditioninformation received by said receiver is to be analyzed by said analyzerand, if said determination result produced by said generator indicatesthat said second user condition information is not to be analyzed bysaid analyzer, said analyzer analyzes only said first user conditioninformation.
 4. The information-processing apparatus according to claim1, further comprising: an output unit configured to output a voice andan image, which are obtained as a synthesis result carried out by saidsynthesizer, wherein said parameter setting unit sets a controlparameter of said output unit on the basis of an analysis result carriedout by said analyzer.
 5. The information-processing apparatus accordingto claim 1 wherein said parameter setting unit sets a control parameterof an electronic apparatus, which is connected to saidinformation-processing apparatus, on the basis of an analysis resultcarried out by said analyzer.
 6. The information-processing apparatusaccording to claim 1, wherein said parameter setting unit sets a controlparameter of said other information-processing apparatus on the basis ofan analysis result carried out by said analyzer, and saidinformation-processing apparatus further comprises a transmitterconfigured to transmit said control parameter set by said parametersetting unit to said other information-processing apparatus.
 7. Theinformation-processing apparatus according to claim 1, furthercomprising: a determination unit configured to generate a determinationresult as to whether or not a control parameter transmitted by saidother information-processing apparatus is to be accepted.
 8. Theinformation-processing apparatus according to claim 1, wherein saidsecond user condition information includes information describingmovement of the other user.
 9. The information-processing apparatusaccording to claim 1, wherein said synthesizer increases a size of saidimage of said other user when said other user leans forward.
 10. Theinformation-processing apparatus according to claim 1, wherein saidsynthesizer increases a concentration of said image of said other userwhen said other user leans forward.
 11. The information-processingapparatus according to claim 1, wherein said synthesizer increases avolume of said voice of said other user when said other user leansforward.
 12. The information-processing apparatus according to claim 1,wherein said second user condition information includes informationdescribing movement of the other user generated from real-time video ofthe other user.
 13. The information-processing apparatus according toclaim 1, wherein said second user condition information includesinformation describing sound volume of said voice of the other user. 14.The information-processing apparatus according to claim 13, wherein saidsynthesizer increases a size of said image of said other user when saidsound volume of said voice of said other user exceeds a threshold. 15.The information-processing apparatus according to claim 14, wherein saidsecond user condition information includes information describing soundvolume of said voice of the other user generated from real-time video ofthe other user.
 16. An information-processing method adopted by aninformation-processing apparatus as a method for communicating with another information-processing apparatus, which is connected to saidinformation-processing apparatus by a network, saidinformation-processing method comprising: reproducing moving videocontent data common to said information-processing apparatus and saidother information-processing apparatus synchronously with said otherinformation-processing apparatus; inputting first user conditioninformation including a voice and image of a user operating saidinformation-processing apparatus; receiving second user conditioninformation including a voice and image of an other user from said otherinformation-processing apparatus operated by said other user;synthesizing a voice and image of said moving video content datasynchronously reproduced at said reproducing with said voice and imageof said other user received at said receiving by superimposing the voiceand image onto the moving video content data, the synthesizing includingcreating a single image including both the voice and image of the movingvideo content data and the voice and image of the other user such thatthe image of the content data extends within an area and at least partof the image of the other user is included within the area of the imageof the moving video content data; simultaneously displaying said voiceand image of said moving video content data synthesized with said voiceand image of said other user; analyzing at least either said first usercondition information input at said inputting or said second usercondition information received at said receiving; and setting a controlparameter used for controlling said synthesizing on the basis of ananalysis result produced at said analyzing.
 17. A recording medium forrecording a program to be executed by a computer to perform a method forcommunicating with an information-processing apparatus, which isconnected to said computer by a network, said method comprising:reproducing content data common to said information-processing apparatusand said other information-processing apparatus synchronously with saidother information-processing apparatus; inputting first user conditioninformation including a voice and image of a user operating saidinformation-processing apparatus; receiving second user conditioninformation including a voice and image of an other user from said otherinformation-processing apparatus operated by said other user;synthesizing a voice and image of said moving video content datasynchronously reproduced at said reproducing with said voice and imageof said other user received at said receiving by superimposing the voiceand image onto the moving video content data, the synthesizing includingcreating a single image including both the voice and image of the movingvideo content data and the voice and image of the other user such thatthe image of the content data extends within an area and at least partof the image of the other user is included within the area of the imageof the moving video content data; simultaneously displaying said voiceand image of said moving video content data synthesized with said voiceand image of said other user; analyzing at least either said first usercondition information input at said inputting or said second usercondition information received at said receiving; and setting a controlparameter used for controlling said synthesizing on the basis of ananalysis result produced at said analyzing.